Application load adaptive multi-stage parallel data processing architecture

ABSTRACT

Systems and methods provide an extensible, multi-stage, realtime application program processing load adaptive, manycore data processing architecture shared dynamically among instances of parallelized and pipelined application software programs, according to processing load variations of said programs and their tasks and instances, as well as contractual policies. The invented techniques provide, at the same time, both application software development productivity, through presenting for software a simple, virtual static view of the actually dynamically allocated and assigned processing hardware resources, together with high program runtime performance, through scalable pipelined and parallelized program execution with minimized overhead, as well as high resource efficiency, through adaptively optimized processing resource allocation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/273,731 filed Sep. 23, 2016, which is a continuation of U.S.application Ser. No. 15/183,860 filed Jun. 16, 2016 (now U.S. Pat. No.9,465,667), which is a divisional of U.S. application Ser. No.15/042,159 filed Feb. 12, 2016 (now U.S. Pat. No. 9,400,694), which is acontinuation of U.S. application Ser. No. 14/261,384 filed Apr. 24, 2014(now U.S. Pat. No. 9,262,204), which is a continuation of U.S.application Ser. No. 13/684,473 filed Nov. 23, 2012 (now U.S. Pat. No.8,789,065), which claims the benefit of the following provisionalapplications:

-   [1] U.S. Provisional Application No. 61/657,708, filed Jun. 8, 2012;-   [2] U.S. Provisional Application No. 61/673,725, filed Jul. 19,    2012;-   [3] U.S. Provisional Application No. 61/721,686, filed Nov. 2, 2012;    and-   [4] U.S. Provisional Application No. 61/727,372, filed Nov. 16,    2012.

This application is also related to the following:

-   [5] U.S. Utility application Ser. No. 13/184,028, filed Jul. 15,    2011;-   [6] U.S. Utility application Ser. No. 13/270,194, filed Oct. 10,    2011 (now U.S. Pat. No. 8,490,111);-   [7] U.S. Utility application Ser. No. 13/277,739, filed Nov. 21,    2011 (now U.S. Pat. No. 8,561,076); and-   [8] U.S. Utility application Ser. No. 13/297,455, filed Nov. 16,    2011.

All above identified applications are hereby incorporated by referencein their entireties.

BACKGROUND Technical Field

This invention pertains to the field of data processing and networking,particularly to techniques for connecting tasks of parallelized programsrunning on multi-stage manycore processor with each other as well aswith external parties with high resource efficiency and high dataprocessing throughput rate.

Descriptions of the Related Art

Traditionally, advancements in computing technologies have fallen intotwo categories. First, in the field conventionally referred to as highperformance computing, the main objective has been maximizing theprocessing speed of one given computationally intensive program runningon a dedicated hardware comprising a large number of parallel processingelements. Second, in the field conventionally referred to as utility orcloud computing, the main objective has been to most efficiently share agiven pool of computing hardware resources among a large number of userapplication programs. Thus, in effect, one branch of computingtechnology advancement effort has been seeking to effectively use alarge number of parallel processors to accelerate execution of a singleapplication program, while another branch of the effort has been seekingto efficiently share a single pool of computing capacity among a largenumber of user applications to improve the utilization of the computingresources.

However, there have not been any major synergies between these twoefforts; often, pursuing any one of these traditional objectives ratherhappens at the expense of the other. For instance, it is clear that apractice of dedicating an entire parallel processor based (super)computer per individual application causes severely sub-optimalcomputing resource utilization, as much of the capacity would be idlingmuch of the time. On the other hand, seeking to improve utilization ofcomputing systems by sharing their processing capacity among a number ofuser applications using conventional technologies will causenon-deterministic and compromised performance for the individualapplications, along with security concerns.

As such, the overall cost-efficiency of computing is not improving asmuch as any nominal improvements toward either of the two traditionalobjectives would imply: traditionally, single application performancemaximization comes at the expense of system utilization efficiency,while overall system efficiency maximization comes at the expense ofperformance of by the individual application programs. There thus existsa need for a new parallel computing architecture, which, at the sametime, enables increasing the speed of executing application programs,including through execution of a given application in parallel acrossmultiple processor cores, as well as improving the utilization of thecomputing resources available, thereby maximizing the collectiveapplication processing throughput for a given cost budget.

Moreover, even outside traditional high performance computing, theapplication program performance requirements will increasingly beexceeding the processing throughput achievable from a single centralprocessing unit (CPU) core, e.g. due to the practical limits beingreached on the CPU clock rates. This creates an emerging requirement forintra-application parallel processing (at ever finer grades) also formainstream software programs (i.e. applications not traditionallyconsidered high performance computing). Notably, these internallyparallelized mainstream enterprise and web applications will be largelydeployed on dynamically shared cloud computing infrastructure.Accordingly, the emerging form of mainstream computing calls fortechnology innovation supporting the execution of large number ofinternally parallelized applications on dynamically shared resourcepools, such as manycore processors.

Furthermore, conventional microprocessor and computer systemarchitectures use significant portions of their computation capacity(e.g. CPU cycles or core capacity of manycore arrays) for handling inputand output (TO) communications to get data transferred between a givenprocessor system and external sources or destinations as well as betweendifferent stages of processing within the given system. For data volumeintensive computation workloads and/or manycore processor hardware withhigh IO bandwidth needs, the portion of computation power spent on IOand data movements can be particularly high. To allow using maximizedportion of the computing capacity of processors for processing theapplication programs and application data (rather than for systemfunctions such as IO data movements), architectural innovations are alsoneeded in the field of manycore processor IO subsystems. In particular,there is a need for a new manycore processor system data flow and IOarchitecture whose operation, while providing high IO data throughputperformance, causes little or no overhead in terms of usage of thecomputation units of the processor.

SUMMARY

The invented systems and methods provide an extensible, multi-stage,application program load adaptive, parallel data processing architectureshared dynamically among a set of application software programsaccording to processing load variations of said programs. The inventedtechniques enable any program task instance to exchange data with any ofthe task instances of its program within the multi-stage parallel dataprocessing platform, while allowing any of said task instances to beexecuting at any core of their local processors, as well allowing anyidentified destination task instance to be not assigned for execution byany core for periods of time, and while said task instances lackknowledge of which core, if any, at said platform is assigned forexecuting any of said task instances at any given time.

An aspect of the invention provides a system for informationconnectivity among tasks of a set of software programs hosted on amulti-stage parallel data processing platform. Such a systemcomprises: 1) a set of manycore processor based processing stages, eachstage providing an array of processing cores, wherein each of said tasksis hosted on one of the processing stages, with tasks hosted on a givenprocessing stage referred to as locally hosted tasks of that stage, 2) ahardware implemented data packet switching cross-connect (XC) connectingdata packets from an output port of a processing stage to an input portof a given processing stage if a destination software program task ofthe data packet is hosted at the given processing stage, and 3) ahardware implemented receive logic subsystem, at any given one of theprocessing stages, connecting data packets from input ports of the givenprocessing stage to the array of cores of that stage, so that a givendata packet is connected to such a core, if any exist at a given time,among said array that is assigned at the given time to process a programinstance to which the given input packet is directed to. Variousembodiments of such systems further comprise features whereby: a) at agiven processing stage, a hardware implemented controller i)periodically allocates the array of cores of the given stage amonginstances of its locally hosted tasks at least in part based on volumesof data packets connected through the XC to its locally hosted tasks andii) accordingly inserts the identifications of the destination programsfor the data packets passed from the given processing stage forswitching at the XC, to provide isolation between different programsamong the set; b) the system supports multiple instances of each of thelocally hosted tasks at their processing stages, and packet switchingthrough the XC to an identified instance of a given destination programtask; c) said tasks are located across at least a certain subset of theprocessing stages so as to provide an equalized expected aggregate taskprocessing load for each of the processing stages of said subset; and/ord) said tasks are identified with incrementing intra-program task IDsaccording to their descending processing load levels within a givenprogram, wherein, among at least a subset of the processing stages, eachprocessing stage of said subset hosts one of the tasks of each of theset programs so as to equalize sums of said task IDs of the taskslocated on each of the processing stages of said subset.

An aspect of the invention further provides a method for informationconnectivity among tasks of a set of software programs. Such a methodcomprises: 1) hosting said tasks on a set of manycore processor basedprocessing stages, each stage providing an array of processing cores,with tasks hosted on a given processing stage referred to as locallyhosted tasks of that stage, 2) at a data packet switching cross-connect(XC), connecting data packets from an output port of a processing stageto an input port of a given processing stage if a destination softwareprogram task identified for a given data packet is hosted at the givenprocessing stage, and 3) at any given one of the processing stages,connecting data packets from input ports of the given processing stageto the array of cores of that stage, so that a given data packet isconnected to such a core, if any exist at a given time, among said arraythat is assigned at the given time to process a program instance towhich the given input packet is directed to. Various embodiments of themethod comprise further steps and features as follows: a) periodicallyallocating, by a controller at a given one of the processing stages, thearray of cores of the given stage among instances of its locally hostedtasks at least in part based on volumes of data packets connectedthrough the XC to its locally hosted tasks, with the controller,according to said allocating, inserting the identifications of thedestination programs for the data packets passed from the givenprocessing stage for switching at the XC, to provide isolation betweendifferent programs among the set; b) the steps of allocating andconnecting, both at the XC and the given one of the processing stages,are implemented by hardware logic that operates without softwareinvolvement; c) supporting multiple instances of each of the locallyhosted tasks at their processing stages, and packet switching throughthe XC to an identified instance of a given destination task; d) saidtasks are located across at least a certain subset of the processingstages so as to provide an equalized expected aggregate task processingload for each of the processing stages of said subset; and/or e) saidtasks are identified with incrementing intra-program task IDs accordingto their descending processing load levels within a given program,wherein, among at least a subset of the processing stages, eachprocessing stage of said subset hosts one of the tasks of each of theset programs so as to equalize sums of said task IDs of the taskslocated on each of the processing stages of said subset.

A further aspect of the invention provides hardware logic system forconnecting input data to instances of a set of programs hosted on amanycore processor having an array of processing cores. Such a systemcomprises: 1) demultiplexing logic for connecting input data packetsfrom a set of input data ports to destination program instance specificinput port buffers based on a destination program instance identifiedfor each given input data packet, and 2) multiplexing logic forconnecting data packets from said program instance specific buffers tothe array of cores based on identifications, for each given core of thearray, of a program instance assigned for execution at the given core atany given time. An embodiment of the system further comprises a hardwarelogic controller that periodically assigns, at least in part based onvolumes of input data packets at the program instance specific inputport buffers, instances of the programs for execution on the array ofcores, and accordingly forms, for the multiplexing logic, theidentification of the program instance that is assigned for execution ateach core of the array of cores.

Yet further aspect of the invention provides a method for connectinginput data to instances of a set of programs hosted on a manycoreprocessor having an array of processing cores. Such a methodcomprises: 1) demultiplexing input data packets from a set of input dataports to destination program instance specific input port buffersaccording to a destination program instance identified for each giveninput data packet, and 2) multiplexing data packets from said programinstance specific buffers to the array of cores according toidentifications, for each given core of the array, of a program instanceassigned for execution at the given core at any given time. In aparticular embodiment of the method comprise a further step as follows:periodically forming the identifications of the program instancesexecuting at the array of cores through i) allocating the array of coresamong the set of programs at least in part based on volumes of inputdata packets at the input port buffers associated with individualprograms of the set and ii) assigning, based at least in part based onsaid allocating, the cores of the array for executing specific instancesof the programs. Moreover, in an embodiment, the above method isimplemented by hardware logic that operates without softwareinvolvement.

A yet further aspect of the invention provides a method for periodicallyarranging a set of executables of a given software program in anexecution priority order, with an executable referring to a task, aninstance, an instance of a task of the program, or equals thereof. Sucha method comprises: 1) buffering input data at an array of executablespecific input port buffers, wherein a buffer within said array buffers,from an input port associated with the buffer, such data that arrivedthat is directed to the executable associated with the buffer, 2)calculating numbers of non-empty buffers associated with each of theexecutables, and 3) ranking the executables in their descendingexecution priority order at least in part according to their descendingorder in terms numbers of non-empty buffers associated with each givenexecutable. In a particular embodiment of this method, the step ofranking involves I) forming, for each given executable, a 1^(st) phasebit vector having as many bits as there are input ports from where thebuffers receive their input data, with this number of ports denoted withX, and wherein a bit at index x of said vector indicates whether thegiven executable has exactly x non-empty buffers, with x being aninteger between 0 and X, II) forming, from bits at equal index values ofthe 1^(st) phase bit vectors of each of the executables, a row of X2^(nd) phase bit vectors, where a bit at index y of the 2^(nd) phase bitvector at index x of said row indicates whether an executable with IDnumber y within the set has exactly x non-empty buffers, wherein y is aninteger from 0 to a maximum number of the executables less 1, as well asIII) the following substeps: i) resetting the present priority orderindex to a value representing a greatest execution priority; and ii)until either all bits of each of the 2^(nd) phase bit vectors arescanned or an executable is associated with the lowest availableexecution priority, scanning the row of the 2^(nd) phase bit vectors foractive-state bits, one 2^(nd) phase bit vector at a time, starting fromrow index X while decrementing the row index after reaching bit index 0of any given 2^(nd) phase bit vector, and based upon encountering anactive-state bit: i) associating the executable with ID equal to theindex of the active-state bit within its 2^(nd) phase bit vector withthe present priority order index and ii) changing the present priorityorder index to a next lower level of execution priority. Moreover, in anembodiment, the above method is implemented by hardware logic thatoperates without software involvement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows, in accordance with an embodiment of the invention, afunctional block diagram for multi-stage manycore processor system.

FIG. 2 shows, in accordance with an embodiment of the invention, afunctional block diagram for a cross-connect at the multi-stage manycoreprocessor system of FIG. 1.

FIG. 3 shows, in accordance with an embodiment of the invention, ahigh-level functional block diagram for any of the manycore processorsystems in the multi-stage processor system in FIG. 1.

FIG. 4 shows, in accordance with an embodiment of the invention, afunctional block diagram for the input data receive logic subsystem forthe manycore processor system per FIG. 3.

FIG. 5 shows, in accordance with an embodiment of the invention, afunctional block diagram for the application load adaptive parallel dataprocessing subsystem for a given manycore processing system of FIG. 3within the multi-stage processor system in FIG. 1.

FIG. 6 illustrates, in accordance with an embodiment of the invention, acontext diagram for the process of mapping (incl. selecting and placing)instances of the locally hosted application tasks to execute on theprocessing cores of the application load adaptive parallel dataprocessing system per FIG. 5.

FIG. 7 illustrates, in accordance with an aspect of the invention, aflow diagram and major steps for the process per FIG. 6.

FIG. 8 illustrates, in accordance with an embodiment of the invention, amemory access architecture for the multi-core fabric of the dataprocessing system per FIG. 5.

FIG. 9 shows, in accordance with an embodiment of the invention, at moredetail level a portion of an embodiment of a logic system per FIG. 8concerning write access from the cores of the fabric to the applicationinstance (app-inst) specific fabric memory segments.

FIG. 10 shows, in accordance with an embodiment of the invention, atmore detail level an embodiment of a portion of a logic system per FIG.8 concerning read access by processing cores within the fabric to theapp-inst specific fabric memory segments.

DETAILED DESCRIPTION

General notes about this specification (incl. text in the drawings):

-   -   For brevity: ‘application (program)’ is occasionally written in        as ‘app’, ‘instance’ as ‘inst’ and ‘application-task/instance’        as ‘app-task/inst’.    -   Receive (RX) direction is toward the cores of the manycore        processor of a given processing stage, and transmit (TX)        direction is outward from the cores.    -   The term IO refers both to the system 1 (FIG. 1) external input        and output ports as well as ports interconnecting the processing        stages 300 of the system.    -   Ports, such as external or inter-stage ports of the multi-stage        parallel processing system 1 (FIG. 1) can be implemented either        as distinct physical ports or as e.g. time or frequency division        channels on shared physical connections.    -   Terms software program, application program, application and        program are used interchangeably in this specification, and each        generally refer to any type of computer software able to run on        data processing systems based on the architecture.    -   Term ‘task’ in this specification refers to a part of a program,        and covers the meanings of related terms such as actor, thread        etc.    -   References to a “set of” units of a given type, such as        programs, logic modules or memory segments can, depending on the        nature of a particular embodiment or operating scenario, refer        to any positive number of such units.    -   While the term ‘processor’ more specifically refers to the        processing core fabric 510 (FIG. 5), it will also be used, where        it streamlines the text, to refer to a processor system 500        (FIGS. 3-4) and a processing stage 300 (FIGS. 1 and 3) within        the system 1.    -   Typically, there will be one task type per an application hosted        per each of the processing stages 300 in the system 1 per FIG. 1        (while the system 1 supports multiple processing stages and        multiple application programs per each stage).    -   A master type task of a single application-instance (app-inst)        hosted at entry stage processing system can have multiple        parallel worker type tasks of same type hosted at multiple        worker stage processing systems. Generally, a single upstream        app-inst-task can feed data units to be processed in parallel by        multiple downstream app-inst-task:s within the same system 1.    -   Identifiers such as ‘master’ and ‘worker’ tasks or processing        stages are not used here in a sense to restrict the nature of        such tasks or processing; these identifiers are here used        primarily to illustrate a possible, basic type of distribution        of workloads among different actors. For instance, the entry        stage processing system may host, for a given application,        simply tasks that pre-process (e.g. qualify, filter, classify,        format, etc.) the RX data units and pass them to the worker        stage processing systems as tagged with the pre-processing        notations, while the worker stage processor systems may host the        actual master (as well as worker) actors conducting the main        data processing called for by such received data units.        Generally, a key idea of the presented processing system and IO        architecture is that the worker stages of processing—where bulk        of the intra-application parallel and/or pipelined processing        typically is to occur, providing the performance gain of using        parallel task instances and/or pipelined tasks to lower the        processing latency and improve the on-time IO throughput—receive        their input data units as directed to specific destination        app-task instances, while the external parties are allowed to        communicate with a given application program hosted on a system        1 through a single, constant contact point (the ‘master’ task        hosted on the entry stage processor, possibly with its specified        instance).    -   Specifications below assume there to be X IO ports, Y core slots        on a processor 500, M application programs configured and up to        N instances per each application for a processor 500, and up to        T tasks (or processing stages) per a given application        (instance), wherein the capacity parameters X, Y, M, N and T are        some positive integers, and wherein the individual ports, cores,        applications, tasks and instances, are identified with their        ID#s ranging from 0 to said capacity parameter value less 1 for        each of the measures (ports, cores, apps, instances, tasks or        processing stages).

The invention is described herein in further detail by illustrating thenovel concepts in reference to the drawings. General symbols andnotations used in the drawings:

-   -   Boxes indicate a functional digital logic module; unless        otherwise specified for a particular embodiment, such modules        may comprise both software and hardware logic functionality.    -   Arrows indicate a digital signal flow. A signal flow may        comprise one or more parallel bit wires. The direction of an        arrow indicates the direction of primary flow of information        associated with it with regards to discussion of the system        functionality herein, but does not preclude information flow        also in the opposite direction.    -   A dotted line marks a border of a group of drawn elements that        form a logical entity with internal hierarchy, such as the        modules constituting the multi-core processing fabric 110 in        FIG. 1.    -   Lines or arrows crossing in the drawings are decoupled unless        otherwise marked.    -   For clarity of the drawings, generally present signals for        typical digital logic operation, such as clock signals, or        enable, address and data bit components of write or read access        buses, are not shown in the drawings.

FIGS. 1-10 and related descriptions below provide specifications forembodiments and aspects of an extensible, multi-stage, applicationprogram load and type adaptive, multi-stage parallel data processingsystem, including for the input and output (IO) subsystems thereof.

FIG. 1 illustrates, according to an embodiment of the invention, amulti-stage manycore processor system architecture, comprising a set ofapplication processing load adaptive manycore processing stagesinterconnected by a packet destination app-task-inst controlled crossconnect. The discussion in the following details an illustrative exampleembodiment of this aspect of the invention. Note that the number ofprocessing stages 300 and XC ports 40 shown is just for a purpose of onepossible example; various implementations may have any practical numberof such stages and ports.

General operation of the application load adaptive, multi-stage paralleldata processing system 1 per FIG. 1, focusing on the main IO data flows,is as follows: The system 1 provides data processing services to be usedby external parties (e.g. client portions of programs whose serverportions run on the system 1) over networks. The system 1 receives dataunits (e.g. messages, requests, data packets or streams to be processed)from its users through its RX network ports 10, and transmits theprocessing results to the relevant parties through its TX network ports50. Naturally the network ports of the system of FIG. 1 can be used alsofor connecting with other (intermediate) resources and services (e.g.storage, data bases etc.) as and if necessary for the system to producethe requested processing results to the relevant external parties. Theapplication program tasks executing on the entry stage manycoreprocessor 300 are typically of ‘master’ type for parallelizedapplications, i.e., they manage and distribute the processing workloadsfor ‘worker’ type tasks running on the worker stage manycore processingsystems 300 (note that the processor system 300 hardware implementationsare similar for all instances of the processing system 300). Theinstances of master tasks typically do preliminary processing (e.g.message/request classification, data organization) and workflowmanagement based on given input packet(s), and then typically involveappropriate worker tasks at their worker stage processors (see FIG. 1for context) to perform the data processing called for by the giveninput packet(s), potentially in the context of and in connection withother related input packets and/or other data elements (e.g. in memoryor storage resources accessible by the system 1) referred to by suchinput packets. (Note that processors 300 can also have access to thesystem memories through interfaces additional to the IO ports shown inthe FIGS.) Accordingly, the master tasks typically pass on the receiveddata units (using direct connection techniques to allow most of the datavolumes being transferred to bypass the actual processor cores) throughthe XC 200 to the worker stage processors, with the destination app-taskinstance identified for each data unit. As a security feature, toprovide isolation among the different applications 620 (FIG. 6)configured to run on the processors 300 of the system 1, by default thehardware controller 540 (FIGS. 5 and 7) of each processor 300, ratherthan any application software (executing at a given processor 300),inserts the application ID# bits for the data packets passed to the XC200. That way, the tasks of any given application running on theprocessing stages 300 in a system 1 can trust that the packets theyreceived from the XC 200 are from its own application. Note that thecontroller 540 determines, and therefore knows, the application ID# thateach given core within its processor 500 is assigned to at any giventime, via the app-inst to core mapping info 560 that the controllerproduces (FIGS. 4, 5 and 7). Therefore the controller 540 is able toinsert the presently-assigned app ID# bits for the inter-task data unitsbeing sent from the cores of its processing stage 300 over thecore-specific output ports 20, 210 (FIG. 3) to the XC 200.

While the processing of any given application (server program) at asystem 1 is normally parallelized and/or pipelined, and involvesmultiple tasks (many of which tasks and instances thereof can executesimultaneously on the manycore arrays of the processors 300), the systemenables external parties to communicate with any such application hostedon the system 1 without having to know about any specifics (incl.existence, status, location) of their internal tasks or parallelinstances thereof. As such, the incoming data units to the system 1 areexpected to identify just their destination application (and where itmatters, the application instance number), rather than any particulartask within it. Moreover, the system enables external parties tocommunicate with any given application hosted on a system 1 through anyof the network ports 10, 50 without knowing whether or at which coresany instance of the given application task (app-task) may be executingat any time. Furthermore, the architecture enables the aforesaidflexibility and efficiency through its hardware logic functionality, sothat no system or application software running on the system 1 needs toeither be aware of whether or where any of the instances of any of theapp-tasks may be executing at any given time, or through which port anygiven inter-task or external communication may have occurred or beoccurring. Thus the system 1, while providing a highly dynamic,application workload adaptive usage of the system processing andcommunications resources, allows the software running on and/or remotelyusing the system to be designed with a straightforward, abstracted viewof the system: the software (both the server programs hosted on a system1 as well as clients etc. remote agents interacting with such programshosted on the system) can assume that all applications (as well alltheir tasks and instances thereof) hosted on by the given system 1 arealways executing on their virtual dedicated processor cores within thesystem. Also, where useful, said virtual dedicated processors can alsobe considered by software to be time-share slices on a single (very highspeed) processor. The architecture thereby enables achieving, at thesame time, both the vital application software development productivity(simple, virtual static view of the actually highly dynamic processinghardware) together with high program runtime performance (scalableparallel program execution with minimized overhead) and resourceefficiency (adaptively optimized resource allocation) benefits.Techniques enabling such benefits of the architecture are described inthe following through more detailed technical study of the system 1 andits sub systems.

In FIG. 1, the processing stage 300 specific XC IO ports 40 contain oneinput and output port per a processing core at any given stage, withsuch individual IO ports of any given stage identified as ports #0, 1, .. . , Y−1 (noting that the input ports of any given processing stage arenot tied to or associated with any particular core, but instead, inputdata units can be connected from all input ports to all cores of anygiven processing stage as needed). The XC 200 provides data unit(referred to as packet) level switched, restriction-free, any-to-anyconnectivity among the mentioned processing stage IO ports of the sameport index #y (y=0, 1, . . . Y−1): E.g. the XC provides packet-switchedconnectivity to input ports #5 of each stage 300 from the output ports#5 of each stage 300 of the system 1 (assuming Y is greater than 5).This cross-connectivity is implemented through data source specificbuffering and load-weigh prioritized fair muxing of packets to the XCoutput ports (i.e. to processing stage 300 input ports 30). Anembodiment of a micro-architecture for such XC output port logic is asillustrated in FIG. 2.

FIG. 2 presents, according to an embodiment of the invention, afunctional block diagram for forming at the XC 200 a given input port290 (see FIG. 3) to a given processor 300 of FIG. 1. The discussion inthe following details an illustrative example embodiment of this aspectof the invention.

The XC 200 subsystems per FIG. 2 provide data connectivity to a giveninput port #y (y=0, 1, . . . Y−1) from output ports #y of each of theprocessing systems 300 of the system 1, and there is a subsystem perFIG. 2 for each input port 290 to each processing system 300. Note thatthe XC 200 is formed by providing the processing stage input port 290specific subsystem per FIG. 2 for each input port of each of theprocessing stages 300 interconnected by the XC 200. At each a subsystemper FIG. 2, there are first-in first-out buffers (FIFOs) 260 per eachpreceding processing stage of the input packets, in which FIFOs packetswhose identified next processing app-task ID matches the processingstage to which the XC output in question connects to (referred to as thelocal processing stage in FIG. 2) are queued, plus an arbitration logicmodule 270 for selecting, at times when a new packet is to be sent overthe local XC output port 290, an appropriate input-stage specific FIFO260 from which to send the next packet to the local processing stage.The next input-stage specific FIFO is chosen by the arbitrator 270 byrunning a round-robin selection algorithm first among those input-stagespecific FIFOs whose fill level is indicated 265 as being above adefined threshold, and in the absence of such FIFOs, running a plainround robin algorithm across all the FIFOs for the given XC output port.For the FIFO module 260 selected by the arbitrator at any given time,the arbitrator activates the read enable signal 271. The arbitrator alsocontrols the mux (mux) 280 to connect to its output 290 the packetoutput 265 from the FIFO module 240 selected at the time.

Note that in FIG. 2, there are submodules 250 and 260 associated withthe input data streams from each of the preceding processing stages #0,1, . . . T−1 similar to those drawn in more detail for the stage #0.Though not included in FIG. 2, similar signals (fill level indication265 and read enable 271) exist between each of the preceding processingstage specific FIFO modules 240 and the arbitrator 270, as is shownbetween the module specific to preceding stage #0 and the arbitrator.

Moreover, the set of applications 610 (FIG. 6) configured to run on thesystem 1 have their tasks identified by (intra-application) IDsaccording to their descending order of relative (time-averaged) workloadlevels. The sum of the intra-application task IDs (each representing theworkload ranking of its tasks within its application) of the app-taskshosted at any given processing system 300 is equalized by appropriatelyconfiguring the tasks of differing ID#s (i.e. of differing workloadlevels) across the applications for each processing system 300, toachieve optimal overall load balancing. For instance, in case of fourprocessing stages 300 (as shown in the example of FIG. 1), if the systemis shared among four applications and each of that set of applicationshas four tasks, for each application of that set, the busiest task (i.e.the worker task most often called for or otherwise causing the heaviestprocessing load among the tasks of the app) is given ID#0, the secondbusiest task ID#1, the third busiest ID#2, and the fourth ID #3. Tobalance the processing loads across the applications among the workerstage processors 300 of the system 1, the worker stage processor #t getstask ID#t+m (rolling over at 3 to 0) of the application ID #m (t=0, 1, .. . T−1; m=0, 1, . . . M−1). In this example scenario of fourapplication streams, four worker tasks per app as well as fourprocessors 300 in a system 1, the above scheme causes the task IDs ofthe set of apps to be placed at the processing stages per the tablebelow (t and m have the meaning per the previous sentence):

App ID# m (to right) Stage# t (below) 0 1 2 3 0 0 1 2 3 1 1 2 3 0 2 2 30 1 3 3 0 1 2

As seen in the example of the table above, the sum of the task ID#s(with each task ID# representing the workload ranking of its task withinits application) is the same for any row i.e. for each of the fourprocessing stages of this example. Applying this load balancing schemefor differing numbers of processing stages, tasks and applications isstraightforward based on the above example and the discussion herein. Insuch system wide processing load balancing schemes supported by system1, a key idea is that each worker stage processor 300 gets one of thetasks from each of the applications so that collectively the tasksconfigured for any given worker stage processor 500 have the intra-apptask IDs of the full range from ID#0 through ID#T−1 with one task ofeach ID# value (wherein the intra-app task ID#s are assigned for eachapp according to their descending busyness level) so that the overalltask processing load is to be, as much as possible, equal across allworker-stage processors 300 of the system 1. Advantages of these schemessupported by systems 1 include achieving optimal utilization efficiencyof the processing resources and eliminating or at least minimizing thepossibility or effects of any of the worker-stage processors 300 formingsystem wide performance bottlenecks. In FIG. 2, each of the logicmodules 250 for forming write enable signal performs the algorithm perabove, thus selecting which packets (based on their destination app-taskID#) to pass to its local FIFO 260 from its associated precedingprocessing stage.

In the following, we continue by exploring the internal structure andoperation of a given processing stage 300, a high level functional blockdiagram for which is shown in FIG. 3.

FIG. 3, presents, according to an embodiment of the invention, a toplevel functional block diagram for any of the manycore processingsystems 300 in the multi-stage parallel processing system in FIG. 1,involving a RX logic subsystem and manycore processor subsystem. Thediscussion in the following details an illustrative example embodimentof this aspect of the invention.

As illustrated in FIG. 3, any of the processing systems 300 of system 1(FIG. 1) has, besides manycore processor system 500 (detailed in FIGS.5-10), an RX logic subsystem 400, which connects input data units(packets) from any of the input ports 290 to any of the processing coresof the manycore processor 500, according at which core their indicateddestination app-task-instance may be executing at any given time.Moreover, the monitoring of the buffered input data load levels pertheir destination app-task instances at the RX logic subsystem 400allows optimizing the allocation of processing core capacity of thelocal manycore processor 500 among the application tasks hosted on thegiven processing system 300. The structure and operation of anembodiment of the RX logic subsystem 400 for the manycore processingsystem per FIG. 3 is detailed below in connection with FIG. 4.

FIG. 4 illustrates, according to an embodiment of the invention, maindata flows of the RX logic subsystem 400, which connects input packetsfrom any of the input ports 290 to any of the processing cores of theprocessor system 500, according to at which core the destinationapp-task instance indicated for any given input may be executing at anygiven time. The discussion below details an illustrative exampleembodiment of this aspect of the invention.

The RX logic connecting the input packets from the input ports 290 tothe local processing cores arranges the data from all the input ports290 according to their indicated destination applications and thenprovides for each core of the manycore processor 500 read access to theinput packets for the app-task instance executing on the given core atany given time. At this point, it shall be recalled that there is oneapp-task hosted per processing stage 500 per each of the applications610 (FIG. 6), while there can be up to Y instances in parallel for anygiven app-task. Since there is one app-task per app per processingstage, the term app-inst in the following, including in and inconnection to FIGS. 4-11, means an instance of an application taskhosted at the processing stage under study.

The main operation of the RX logic shown in FIG. 4 is as follows: Firstinput packets arriving over the network input ports 290 are grouped to aset of destination application specific FIFO modules 420, whose filllevels (in part) drives the allocation and assignment of cores at thelocal manycore processor 500 among instances of the app-tasks hosted onthat processor, in order to maximize the total (value-add, e.g. revenue,of the) data processing throughput across all the application programsconfigured for the manycore processor system. From the app-inst specificbuffers 415 within the destination application buffer modules 420, theinput packets are then connected 450 to specific cores of the processor500 where their associated app-inst:s are executing at a given time(when the given app-inst is selected for execution). At greater level ofdetail, the data flow of the RX logic 400, and its interactions with itslocal manycore processor 500, are detailed in the following:

The input packets arriving over the input ports are demuxed byindividual RX network port specific demultiplexers (demux:s) 405 totheir indicated (via overhead bits) destination app-inst and input portspecific FIFO buffers 410. At the RX subsystem 400, there will thus beFIFOs 410 specific to each input port 290 for each app-inst able to runon the manycore processor 500. In FIG. 4, the app-inst specificcollections 415 and application-scope collections 420 of these FIFOs 410is shown for the application ID #1 to keep the diagram reasonablysimple; however similar arrangements exist for each of the applicationsIDs #0 through #N. Similarly, though FIG. 4 for clarity shows theconnections from the input port #1 to the application FIFOs 425, andconnections from the input ports just to application #1 FIFOs, theseconnections shall be understood to exist between each input port 290 andRX FIFO collection 420 of each application. A reason for thesecollections of input port specific buffers 410 for each app-inst is toallow writing all input packets directly, without delaying or blockingother data flows, to a buffer, even when a given destination app-instwas receiving data from multiple, and up to all, of the input ports atthe same time. Moreover, the app-inst level connection of packetsbetween the processing stages 300 (enabled in part by the app-task-instspecific buffering 415) also allows the system 1 to efficiently maintaincontinued data flows across the system specific to particular instancesof application tasks originating or consuming a given sequence of datapackets.

Logic at each application scope FIFO module 420 signals 430 to themanycore processor system 500 the present processing load level of theapplication as a number of the ready to execute instances of the givenapp-task and, as well as the priority order of such instances. Anapp-inst is taken as ready to execute when it has unread input data inits FIFO 410. As discussed in greater depth in connection with FIGS.5-7, based on the info 430 from the applications, the processor system500 periodically, e.g. at intervals of 1024 processor clock cycles,assigns to each of its cores one of the locally hosted app-inst:s, in amanner as to maximize the system wide (value add of the) data processingthroughput. According to such periodic assignments, the processor system500 provides control for the mux:s 450 to connect to each of its coresthe read data bus 440 from the appropriate app-inst FIFO 415. Logic atapp-inst FIFO module 415 selects (at packet boundaries) one of its theport specific FIFOs 410 for reading out data to its associated mux atmodule 450 at times when the given app-inst is selected to execute.Similar FIFO read selection algorithm is used in this case as what wasdescribed in connection to FIG. 2 for selecting a FIFO for reading ontoa port 290. In addition, the controller 540 also dynamically controlsmux:s 580 (FIG. 5) to appropriately connect input data read controlinformation 590 to the app-instance FIFOs 415, to direct reading ofinput data by the app-inst selected to execute on any of its cores atthe given time.

For the info flow 430 (FIGS. 4 and 5), which is used for optimallyallocating and assigning the cores of the processor 500 among thelocally hosted app inst:s, the number of ready to execute instances fora given app-task is taken as its number of FIFO modules 415 that at thegiven time have one or more of their input port specific FIFOs 410non-empty. Moreover, the logic at each app-scope FIFO module 420 ranksits instances in an execution priority order (for the info flow 430)based on how many non-empty FIFOs 410 each of its instance-scope modules415 has. This logic forms, from the modules 415, X instances (equal tonumber of input ports) of N-bit vectors wherein the bit[n] of suchvector instance #x (=0, 1, . . . X) represents whether app-instance #nat the time has (no more and no less than) x non-empty FIFOs 410. Attimes of writing 430 the updated app-inst priority lists to the localmanycore processor system 500, this logic at module 420 scans thesevectors for active bits, starting from priority 0 (highest priority),and proceeding toward greater instance priority index (signifyingdescending instance priority), and from the maximum value of x (that is,X and proceeding down toward 0). When this logic encounters an activebit, the logic writes the ID# number of its associated app-inst (i.e.,the index of that bit, n) to the current priority index at the(descending) priority-indexed app-inst ID# look-up-table (see a formatfor the LUT at Table 3 shown later in this specification, under heading“Summary of process flow and information formats . . . ”), at thecontroller module (540, FIGS. 5 and 7) of the manycore processor system500, for the controller 540 to use when selecting the instances of thegiven application to execute on the cores allocated to that applicationon the following core allocation period. Furthermore, the abovediscussed logic at the any given app-scope FIFO module 420 starts itssuccessive runs of the app-inst priority list production from arevolving bit index n (incrementing by one after each run of thealgorithm, from 0 through N−1 and rolling over to 0 and so forth), toover time provide equality among the instances of the given application(having same number of non-empty port FIFOs 410).

The RX logic subsystem 400 is implemented by digital hardware logic andis able to operate without software involvement. Note that the conceptof software involvement as used in this specification relates to active,dynamic software operation, not to configuration of the hardwareelements according aspects and embodiments of the invention throughsoftware where no change in such configuration is needed to accomplishthe functionality according to this specification.

This specification continues by describing the internal elements andoperation of the processor system 500 (for the processing system 300 ofFIG. 3, within the multi-stage parallel processing system 1 of FIG. 1),a block diagram for an embodiment of which is shown in FIG. 5.

FIG. 5 presents, according to an embodiment of the invention, afunctional block diagram for the manycore processor system 500dynamically shared among instances of the locally hosted applicationprogram tasks, with capabilities for application processing loadadaptive allocation of the cores among the applications, as well as for(as described in relation to FIGS. 8-10) accordant dynamicallyreconfigurable memory access by the app-task instances. The discussionbelow details an illustrative example embodiment of this aspect of theinvention.

Any of the cores 520 of a system 500 can comprise any types of softwareprogram processing hardware resources, e.g. central processing units(CPUs), graphics processing units (GPUs), digital signal processors(DSPs) or application specific processors (ASPs) etc., and inprogrammable logic (FPGA) implementation, the core type for any coreslot 520 is furthermore reconfigurable per expressed demands 430 of theactive app-tasks.

As illustrated in FIG. 5, the processor system 500 comprises an array515 of processing cores 520, which are dynamically shared among a thelocally hosted tasks of a set of application programs configured to runon the system 1. The logic at application specific modules 420 (FIG. 4)write via info flows 430 their associated applications' capacity demandindicators 530 to the controller 540. Each of these indicators 530,referred to herein as core-demand-figures (CDFs), express how many cores520 their associated app-task is presently able utilize for its ready toexecute instances. Moreover, the RX logic for the individualapplications write the application CDFs to a look-up-table (LUT) at thecontroller per Table 1 format, as described later on in thisspecification under heading “Summary of process flow and informationformats . . . ”. Furthermore, these capacity demand expressions 430,written to controller 540 by the RX logic (at module 420) of eachlocally hosted app-task, include a list 535 identifying its readyinstances in a priority order per LUT of Table 3 format, also describedlater on in this specification under the heading “Summary of processflow and information formats . . . ”.

A hardware logic based controller module 540 within the processor system500, through a periodic process, allocates and assigns the cores 520 ofthe processor 500 among the set of applications 610 (FIG. 6) and theirinstances, at least in part based on the CDFs 530 of the applications.This application instance to core assignment process 700 (see FIGS. 6and 7) is exercised periodically, e.g. at intervals such as once per adefined number (for instance 64, 256 or 1024, or so forth) of processingcore clock or instruction cycles. The application instance to coreassignment algorithms of the controller 540 produce, for the applicationinstances on the processor 500, identification 550 of their executioncores (if any, at any given time), as well as for the cores of thefabric 515, identification 560 of their respective app-inst:s toprocess. As shown in FIGS. 4 and 5, the app-inst to core mapping info560 also directs the muxing 450 of input data from an appropriateapp-inst to each core of the array 515. The app-inst to core mappinginfo 550 is also used to configure the muxing 580 of the input data readcontrol signals from the core array 515 (via info flow 590) to the FIFOs415 of the app-inst assigned for any given core.

Note that the verb “to assign” is used herein reciprocally, i.e., it canrefer, depending on the perspective, both to assignment of cores 520 toapp-inst:s 640 (see FIG. 6) as well as to mapping of app-inst:s 640 tocores 520. This is due to that the allocation and mapping algorithms ofthe controller 540 cause one app-inst 640 to be assigned per any givencore 520 of the array 515 by each run of such algorithms 700 (see FIGS.6 and 7). As such, when it is written here, e.g., that a particular core#x is assigned to process a given app-inst #y, it could have also beensaid that app-inst #y is assigned for processing by core #x. Similarly,references such as “core #x assigned to process app-inst #y”, could bewritten in the (more complex) form of “core #x for processing app-inst#y assigned to it”, and so forth.

The controller module 540 is implemented by digital hardware logicwithin the system, and the controller exercises its repeatingalgorithms, including those of process 700 per FIGS. 6-7, withoutsoftware involvement.

FIG. 6 illustrates, according to an embodiment of the invention, contextfor the process 700 performed by the controller logic 540 of the system500, repeatedly selecting and placing the to-be-executing instances 640of the set of locally hosted app-tasks 610 to their assigned targetcores 520 within the array 515. The discussion below details anillustrative example embodiment of this aspect of the invention.

Per FIG. 6, each individual app-task 620 configured for a system 500 hasits collection 630 of its instances 640, even though for clarity ofillustration in FIG. 6 this set of instances is shown only for one ofthe applications within the set 610 configured for a given instance ofsystem 500. Recalling that this multi-stage parallel processingarchitecture is designed for one task per application program perprocessing stage, in the following discussion (incl. text in FIGS. 7-10)of internal aspects of any of the processor systems 500 at a multi-stageprocessor system 1, references to ‘application’ (app) have the meaningof a locally hosted application task (app-task).

Note also that, among the applications 620 there can be supervisory ormaintenance software programs for the system 500, used for instance tosupport configuring other applications 620 for the system 500, as wellas provide general functions such as system boot-up and diagnostics.

In the context of FIGS. 4-6, FIG. 7 provides a data flow diagram for anembodiment of the process 700, which periodically selects app-inst:s forexecution, and places each selected-to-execute app-inst 640 within thesets 630 to one of the cores 520 within the array 515.

FIG. 7 presents, according to an embodiment of the invention, majorphases of the app-inst to core mapping process 700, used for maximizingthe (value-add of the) application program processing throughput of themanycore fabric 510 shared among a number of software programs. Thediscussion below details an illustrative example embodiment of thisaspect of the invention.

The process 700, periodically selecting and mapping the to-be-executinginstances of the set 610 of applications to the array of processingcores within the processor 500, involves the following steps:

-   (1) allocating 710 the array 515 of cores among the set of    applications 610, based on CDFs 530 and CEs 717 of the applications,    to produce for each application 620 a number of cores 520 allocated    to it 715 (for the time period in between the current and the next    run of the process 700); and-   (2) based at least in part on the allocating 710, for each given    application that was allocated at least one core: (a) selecting 720,    according to the app-inst priority list 535, the highest priority    instances of the given application for execution corresponding to    the number of cores allocated to the given application, and (b)    mapping 730 each selected app-inst to one of the available cores of    the array 515, to produce, i) per each core of the array, an    identification 560 of the app-inst that the given core was assigned    to, as well as ii) for each app-inst selected for execution on the    fabric 515, an identification 550 of its assigned core.    The periodically produced and updated outputs 550, 560 of the    controller 540 process 700 will be used for periodically    re-configuring connectivity through the mux:s 450 (FIG. 4) and 580    (FIG. 5) as well as the fabric memory access subsystem 800, as    described in the following with references to FIGS. 8-10.

FIGS. 8-10. and related specifications below describe embodiments of theon-chip memory access subsystem 800 of a manycore processor 500providing non-blocking processing memory access connectivity (incl. forprogram instructions and interim processing results) between theapp-inst:s assigned to cores of the array 515 and app-inst specificmemories at the memory array 850. The manycore fabric memory accesssubsystem per FIGS. 8-10 comprises hardware logic, and is able tooperate without software involvement. The capabilities per FIGS. 8-10provide logic, wiring, memory etc. system resource efficient support forexecuting any app-inst 640 at any core 520 within the processor 500 atany given time (as controlled by the controller 540 that periodicallyoptimizes the allocation and assignment of cores of the array 515 amongthe locally hosted app-inst:s 620), while keeping each given app-instconnected to its own (program instruction and interim processing resultscontaining) memory element at memory array 850.

Fabric Memory Access Subsystem for Manycore Processor Per FIG. 5:

FIG. 8 presents, according to an embodiment of the invention, logicarrangements to provide access by app-inst:s executing at the core arrayto app-inst specific memory locations within the core fabric. Thediscussion below details an illustrative example embodiment of thisaspect of the invention.

Per FIG. 8, to direct write and read control access from the array ofcores 515 to the array of app-inst specific memories 850, the controller540 identifies 550, for a cross-connect (XC) 830 between the core array515 and memory array 850, the presently active source core for write andread control access 810, 840 to each given app-inst specific segment 950within the memory array 850. Similarly, to direct read access by thearray of cores 515 to the array of app-inst specific memories 850, thecontroller also identifies 560 for the XC 870 the memory segment 950 (atthe memory array 850) of the app-inst presently assigned for each givencore 520 of the array.

Based on the control 560 by the controller 540 for a given coreindicating that it will be subject to an app-inst switchover, thecurrently executing app-inst is made to stop executing and itsprocessing state from the core is backed up 810, 940 (FIGS. 8 and 9) tothe segment 950 of that exiting app-inst at the memory array 850 (FIGS.8 and 9), while the processing state of the next instance assigned toexecute on the given core is retrieved 1010, 880 to the core from thememory array 850 (FIGS. 8 and 10). Note that ‘processing state’ hereinrefers to processing status data, if any, stored at the core 520, suchas the current executing app-inst specific processor register filecontents etc. interim processing results. During these app-instswitching proceedings the operation of the cores subject to instanceswitchover is controlled through the controller 540 and switchover logicat the cores 520, with said switchover logic backing up and retrievingthe outgoing and incoming app-inst processing states from the memories850. Cores not indicated by controller 540 as being subject to instanceswitchover continue their processing uninterruptedly through the CoreAllocation Period (CAP) transition times.

Note that applying of updated app-inst ID# configurations 560 for thecore specific mux:s 1020 of XC 870 (see FIGS. 8 and 10), as well asapplying of the updated processing core ID# configurations 550 for theapp-inst specific mux:s 910 at XC 830 (see FIGS. 8 and 9), can be safelyand efficiently done on one mux at a time basis (reducing the systemhardware and software implementation complexity and thus improvingcost-efficiency), since none of the app-inst:s needs to know whether orat which core itself or any other app-inst is executing within thesystem 1 at any given time. Instead of relying on knowledge of the theirrespective previous, current (if any at any given time) or futureexecution cores by either the app-task instances or any system software,the architecture enables flexibly running any instance of any app-taskat any core of the processing systems 300 that they are hosted on.

FIG. 9 shows, according to an embodiment of the invention, at a moredetail level, a portion of the logic system 800 (see FIGS. 5 and 8 forcontext) for providing write access and read access control from thecores of the system 500 to the memories 950 specific to their presentlyassigned execution app-inst:s. The discussion below details anillustrative example embodiment of this aspect of the invention.

The XC 830 comprises a set of app-inst specific mux:s 910, each of whichselects the write and read control access bus from the set 810identified 550 to it for write direction access 940 to its associatedapp-inst specific segment 950 at the memory array 850. Each suchapp-inst specific mux 910 makes these selections based on control 550from the controller 540 that identifies the core (if any) presentlyassigned to process its associated app-inst.

At digital logic design level, the write access (incl. read control) businstance within the set 810 from the core ID #y (y is an integer between0 and Y−1) is connected to the data input #y of each mux 910 of XC 830,so that the identification 550 of the appropriate source core ID# by thecontroller to a given mux 910 causes the XC 830 to connect the write andread control buses 810 from the core array 515 to the proper app-instspecific segments 950 within the memory 850. The controller 540 usesinformation from an application instance ID# addressed look-up-table perTable 4 format (shown later in this specification, under heading“Summary of process flow and information formats . . . ”) in supplyingthe present processing core (if any) identifications 550 to theapplication instance specific mux:s 910 of XC 830 (the info flow 550also includes a bit indicating whether a given app-inst was selected forexecution at a given time—if not this active/inactive app-inst indicatorbit causes the muxes 910 to disable write access to such app-inst'smemory 950).

In addition to write data, address and enable (and any other relevantwrite access signals), the buses 810 and 940 include the read accesscontrol signals including the read address to memory 950, from theirsource cores to their presently assigned processing app-inst:s' memorysegments 950, to direct read access from the cores of the array 515 tothe memory array 850, which function is illustrated in FIG. 10.

FIG. 10 shows, according to an embodiment of the invention, at a greaterlevel of detail a portion of the logic system per FIG. 8 for connectingto each given processing core within a system 500 (FIG. 5) the read databus from the memory 950 specific to the app-inst assigned to any givencore at any given time. The discussion below details an illustrativeexample embodiment of this aspect of the invention.

The XC 870 (see FIG. 8 for context) comprises core specific mux:s 1020,each of which selects the read data bus (from set 1010) of the app-instpresently identified 560 for processing by the core associated with agiven mux 1020 for connection 880 to that core 520.

Similar to the digital logic level description of the mux 910 (inconnection to FIG. 9), the logic implementation for functionalityillustrated in FIG. 10, is such that the read data bus instance (fromset 1010) associated with application instance ID #m (m is an integerbetween 0 and M−1) is connected to the data input #m of each mux 1020instance, so that the identification (by the controller 540) of theactive application instance ID#560 for each of these core specific mux:s1020 of XC 870 causes the XC 870 to connect each given core 520 of thearray 515 in read direction to the memory segment 950 (at memory array850) that is associated with its indicated 560 active app-inst. Thecontroller 540 uses information from a core ID# addressed look-up-tableper Table 5 format (shown in later in this specification under theheading “Summary of process flow and information formats . . . ”) insupplying the active application instance identifications 560 to thecore specific mux:s 1020 of XC 870.

Module-Level Implementation Specifications for the Application Instanceto Core Placement Process:

The steps of the process 700 (FIG. 7), according to an embodiment of theinvention, are described in the following. The process 700 isimplemented by hardware logic in the controller module 540 of aprocessor 500 per FIG. 5. Similar processes 700 are run (independently)for each of the processing stages 300 of a given system 1.

Objectives for the core allocation algorithm 710 include maximizing theprocessor 500 core utilization (i.e., generally minimizing, and so longas there are ready app-inst:s, eliminating core idling), while ensuringthat each application gets at least up to its entitled (e.g. a contractbased minimum) share of the processor 500 core capacity whenever it hasprocessing load to utilize such amount of cores. Each applicationconfigured for a given manycore processor 500 is specified its entitledquota 717 of the cores, at least up to which quantity of cores it is tobe allocated whenever it is able to execute on such number of cores inparallel; sum of the applications' core entitlements (CEs) 717 is not toexceed the total number of core slots in the given processor 500. Eachapplication program on the processor 500 gets from each run of thealgorithm 710:

-   (1) at least the lesser of its (a) CE 717 and (b) Core Demand Figure    (CDF) 530 worth of the cores (and in case (a) and (b) are equal, the    ‘lesser’ shall mean either of them, e.g. (a)); plus-   (2) as much beyond that to match its CDF as is possible without    violating condition (1) for any application on the processor 500;    plus-   (3) the application's even division share of any cores remaining    unallocated after conditions (1) and (2) are satisfied for all    applications 610 sharing the processor 500.

The algorithm 710 allocating cores 520 to application programs 620 runsas follows:

-   -   (i) First, any CDFs 530 by all application programs up to their        CE 717 of the cores within the array 515 are met. E.g., if a        given program #P had its CDF worth zero cores and entitlement        for four cores, it will be allocated zero cores by this step        (i). As another example, if a given program #Q had its CDF worth        five cores and entitlement for one core, it will be allocated        one core by this stage of the algorithm 710. To ensure that each        app-task will be able at least communicate with other tasks of        its application at some defined minimum frequency, the step (i)        of the algorithm 710 allocates for each application program,        regardless of the CDFs, at least one core once in a specified        number (e.g. sixteen) of process 700 runs.    -   (ii) Following step (i), any processing cores remaining        unallocated are allocated, one core per program at a time, among        the application programs whose demand 530 for processing cores        had not been met by the amounts of cores so far allocated to        them by preceding iterations of this step (ii) within the given        run of the algorithm 710. For instance, if after step (i) there        remained eight unallocated cores and the sum of unmet portions        of the program CDFs was six cores, the program #Q, based on the        results of step (i) per above, will be allocated four more cores        by this step (ii) to match its CDF.    -   (iii) Following step (ii), any processing cores still remaining        unallocated are allocated among the application programs evenly,        one core per program at time, until all the cores of the array        515 are allocated among the set of programs 610. Continuing the        example case from steps (i) and (ii) above, this step (iii) will        allocate the remaining two cores to certain two of the programs        (one for each). Programs with zero existing allocated cores,        e.g. program #P from step (i), are prioritized in allocating the        remaining cores at the step (iii) stage of the algorithm 710.

Moreover, the iterations of steps (ii) and (iii) per above are startedfrom a revolving application program ID#s within the set 610, e.g. sothat the application ID# to be served first by these iterations isincremented by one (and returning to ID#0 after reaching the highestapplication ID#) for each successive run of the process 700 and thealgorithm 710 as part of it. Furthermore, the revolving start app ID#sfor the steps (ii) and (iii) are kept at offset from each other equal tothe number of app:s sharing the processor divided by two.

Accordingly, all cores 520 of the array 515 are allocated on each run ofthe related algorithms 700 according to applications processing loadvariations while honoring their contractual entitlements. The allocatingof the array of cores 515 by the algorithm 710 is done in order tominimize the greatest amount of unmet demands for cores (i.e. greatestdifference between the CDF and allocated number of cores for any givenapplication 620) among the set of programs 610, while ensuring that anygiven program gets at least its entitled share of the processing coresfollowing such runs of the algorithm for which it demanded 530 at leastsuch entitled share 717 of the cores.

To study further details of the process 700, let us consider the coresof the processor 500 to be identified as core #0 through core #(Y−1).For simplicity and clarity of the description, we will from hereonconsider an example processor 500 under study with a relatively smallnumber Y of sixteen cores. We further assume here a scenario ofrelatively small number of also sixteen application programs configuredto run on that processor 500, with these applications identified for thepurpose of the description herein alphabetically, as application #Athrough application #P. Note however that the architecture presents noactual limits for the number of cores, applications or their instancesfor a given processor 500. For example, instances of processor 500 canbe configured a number of applications that is lesser or greater than(as well as equal to) the number of cores.

Following the allocation 710 of the set of cores 515 among theapplications 610, for each active application on the processor 500 (thatwere allocated one or more cores by the latest run of the coreallocation algorithm 710), the individual ready-to-execute app-inst:s640 are selected 720 and mapped 730 to the number of cores allocated tothe given application. One schedulable 640 app-inst is assigned per onecore 520 by each run of the process 700.

The app-inst selection 720 step of the process 700 produces, for eachgiven application of the set 610, lists 725 of to-be-executingapp-inst:s to be mapped 730 to the subset of cores of the array 515.Note that, as part of the periodic process 700, the selection 720 ofto-be-executing app-inst for any given active application (such that wasallocated 710 at least one core) is done, in addition to following of achance in allocation 710 of cores among applications, also following achange in app-inst priority list 535 of the given application, includingwhen not in connection to reallocation 710 of cores among theapplications. The active app-inst to core mapping 730 is done logicallyindividually for each application, however keeping track of which coresare available for any given application (by first assigning for eachapplication their respective subsets of cores among the array 515 andthen running the mapping 730 in parallel for each application that hasnew app-inst:s to be assigned to their execution cores).

The app-inst to core mapping algorithm 730 for any application begins bykeeping any continuing app-inst:s, i.e., app-inst:s selected to run onthe array 515 both before and after the present app-inst switchovers,mapped to their current cores also on the next allocation period. Afterthat rule is met, any newly selected app-inst:s for the application aremapped to available cores. Specifically, assuming that a givenapplication was allocated k (a positive integer) cores beyond those usedby its continuing app-inst:s, k highest priority ready butnot-yet-mapped app-inst:s of the application are mapped to k nextavailable (i.e. not-yet-assigned) cores within the array 515 allocatedto the application. In case that any given application had less than kready but not-yet-mapped app-inst:s, the highest priority other (e.g.waiting, not ready) app-inst:s are mapped to the remaining availablecores among the number cores allocated to the given application; theseother app-inst:s can thus directly begin executing on their assignedcores once they become ready. The placing of newly selected app-inst:s,i.e., selected instances of applications beyond the app-inst:scontinuing over the switchover transition time, is done by mapping suchyet-to-be-mapped app-inst:s in incrementing app-inst ID# order toavailable cores in incrementing core ID# order.

Summary of Process Flow and Information Formats Produced and Consumed byMain Stages of the App-Inst to Core Mapping Process:

According to an embodiment of the invention, the production of updatedmappings 560, 550 between selected app-inst:s 725 and the processingcore slots 520 of the processor 500 by the process 700 (FIG. 7,implemented by controller 540 in FIG. 5) from the Core Demand Figures(CDFs) 530 and app-inst priority lists 535 of the applications 620 (FIG.6), as detailed above with module level implementation examples,proceeds through the following stages and intermediate results (inreference to FIG. 7):

The RX logic 400 produces for each application 620 its CDF 530, e.g. aninteger between 0 and the number of cores within the array 515expressing how many concurrently executable app-inst:s 640 theapplication presently has ready to execute. The information format 530,as used by the core allocation phase of the process 700, is such thatlogic with the core allocation module 710 repeatedly samples theapplication CDF bits written 430 to it by the RX logic 400 (FIGS. 4, 5and 7) and, based on such samples, forms an application ID-indexed table(per Table 1 below) as a ‘snapshot’ of the application CDFs as an inputfor next exercising of the process 700. An example of such format of theinformation 530 is provided in Table 1 below—note however that in thehardware logic implementation, the application ID index, e.g. for rangeA through P, is represented by a digital number, e.g., in range 0through 15, and as such, the application ID # serves as the index forthe CDF entries of this array, eliminating the need to actually storeany representation of the application ID for the table providinginformation 530:

TABLE 1 Application ID index CDF value A 0 B 12  C 3 . . . . . . P 1

Regarding Table 1 above, note that the values of entries shown aresimply examples of possible values of some of the application CDFs, andthat the CDF values of the applications can change arbitrarily for eachnew run of the process 700 and its algorithm 710 using snapshots of theCDFs.

Based (in part) on the application ID# indexed CDF array 530 per Table 1above, the core allocation algorithm 710 of the process 700 producesanother similarly formatted application ID indexed table, whose entries715 at this stage are the number of cores allocated to each applicationon the processor 500, as shown in Table 2 below:

TABLE 2 Application ID index Number of cores allocated A 0 B 6 C 3 . . .. . . P 1

Regarding Table 2 above, note again that the values of entries shown aresimply examples of possible number of cores allocated to some of theapplications after a given run on the algorithm 710, as well as that inhardware logic this array 715 can be simply the numbers of coresallocated per application, as the application ID# for any given entry ofthis array is given by the index # of the given entry in the array 715.

The app-inst selection sub-process 720, done individually for eachapplication of the set 610, uses as its inputs the per-application coreallocations 715 per Table 2 above, as well as priority ordered lists 535of ready app-inst IDs of any given application. Each such applicationspecific list 535 has the (descending) app-inst priority level as itsindex, and, as a values stored at each such indexed element, theintra-application scope instance ID#, plus, for processors 500supporting reconfigurable core slot, an indication of the target coretype (e.g. CPU, DSP, GPU or a specified ASP) demanded by the app-inst,as shown in the example of Table 3 below:

TABLE 3 App-inst ID # (identifies the App-inst priority index # --app-inst-specific Target core type application internal (lower memory950 (e.g., 0 denotes CPU, index value signifies more within the 1denotes DSP, and urgent app-inst) memory array 850) 2 denotes GPU, etc.)0 0 0 1 8 2 2 5 2 . . . . . . 15  2 1

Notes regarding implicit indexing and non-specific examples used forvalues per Tables 1-2 apply also for Table 3.

The RX logic 400 writes 430 for each application 620 of the set 610 theintra-app instance priority list 535 per Table 3 to controller 540, tobe used as an input for the active app-inst selection sub-process 720,which produces per-application listings 725 of selected app-inst:s,along with their corresponding target core types where applicable. Basedat least in part on the application specific active app-inst listings725, the core to app-inst assignment algorithm module 730 produces acore ID# indexed array 550 indexed with the application and instanceIDs, and provides as its contents the assigned processing core ID (ifany), per Table 4 below:

TABLE 4 Instance ID (within Processing core ID the application of (value‘Y’ here indicates that column the given app-inst is not Application ID-- to the left) -- LSBs of presently selected for MSBs of index indexexecution at any of the cores) A 0 0 A 1 Y . . . . . . A 15  3 B 0 1 B 1Y . . . . . . B 15  7 C 0 2 . . . . . . . . . P 0 15  . . . . . . P 15 Y

Finally, by inverting the roles of index and contents from Table 4, anarray 560 expressing to which app-inst ID# each given core of the fabric510 got assigned, per Table 5 below, is formed. Specifically, Table 5 isformed by using as its index the contents of Table 4 i.e. the core IDnumbers (other than those marked ‘Y’), and as its contents the app-instID index from Table 4 corresponding each core ID# (along with, whereapplicable, the core type demanded by the given app-inst, with the coretype for any given selected app-inst being denoted as part of theinformation flow 725 (FIG. 7) produced from a data array per Table 3).This format for the app-inst to core mapping info 560 is illustrated inthe example below:

TABLE 5 Core type Instance ID (e.g., 0 denotes CPU, Core ID Application(within the application of 1 denotes DSP, and index ID column to theleft) 2 denotes GPU, etc.) 0 P 0 0 1 B 0 0 2 B 8 2 . . . . . . . . . . .. 15  N 1 1

Regarding Tables 4 and 5 above, note that the symbolic application IDs(A through P) used here for clarity will in digital logic implementationmap into numeric representations, e.g. in the range from 0 through 15.Also, the notes per Tables 1-3 above regarding the implicit indexing(i.e., core ID for any given app-inst ID entry is given by the index ofthe given entry, eliminating the need to store the core IDs in thisarray) apply for the logic implementation of Tables 4 and 5 as well.

In hardware logic implementation the application and the intra-app-instIDs of Table 5 are bitfields of same digital entry at any given index ofthe array 560; the application ID bits are the most significant bits(MSBs) and the app-inst ID bits the least significant (LSBs), andtogether these identify the active app-inst's memory 950 in the memoryarray 850 (for the core with ID# equaling the given index to app-instID# array per Table 5).

By comparing Tables 4 and 5 above, it is seen that the informationcontents at Table 4 are the same as at Table 5; the difference inpurposes between them is that while Table 5 gives for any core slot 520its active app-inst ID#560 to process (along with the demanded coretype), Table 4 gives for any given app-inst its processing core 550 (ifany at a given time). As seen from FIGS. 8-10, the Table 5 outputs areused to configure the core specific mux:s 1020 at XC 870, while theTable 4 outputs are used to configure the app-inst specific mux:s 910 atXC 830.

Note further that, according to the process 700, when the app-inst tocore placement module 730 gets an updated list of selected app-inst:s725 for one or more applications 620 (following a change in either orboth of core to application allocations 715 or app-inst priority lists535 of one or more applications), it will be able to identify fromTables 4 and 5 the following:

-   -   I. The set of activating, to-be-mapped, app-inst:s, i.e.,        app-inst:s within lists 725 not mapped to any core by the        previous run of the placement algorithm 730. This set I is        produced by taking those app-inst:s from the updated selected        app-inst lists 725, per Table 4 format, whose core ID# was ‘Y’        (indicating app-inst not active) in the latest Table 4;    -   II. The set of deactivating app-inst:s, i.e., app-inst:s that        were included in the previous, but not in the latest, selected        app-inst lists 725. This set II is produced by taking those        app-inst:s from the latest Table 4 whose core ID# was not ‘Y’        (indicating app-inst active) but that were not included in the        updated selected app-inst lists 725; and    -   III. The set of available cores, i.e., cores 520 which in the        latest Table 5 were assigned to the set of deactivating        app-inst:s (set II above).        The placer module 730 uses the above info to map the active        app-inst:s to cores of the array in a manner that keeps the        continuing app-inst:s executing on their present cores, thereby        maximizing utilization of the core array 515 for processing the        user applications 620. Specifically, the placement algorithm 730        maps the individual app-inst:s 640 within the set I of        activating app-inst:s in their increasing app-inst ID# order for        processing at core instances within the set III of available        cores in their increasing core ID# order.

Moreover, regarding placement of activating app-inst:s (set I asdiscussed above), the placement algorithm 730 seeks to minimize theamount of core slots for which the activating app-inst demands adifferent execution core type than the deactivating app-inst did. I.e.,the placer will, to the extent possible, place activating app-inst:s tosuch core slots where the deactivating app-inst had the same executioncore type. E.g., activating app-inst demanding the DSP type executioncore will be placed to the core slots where the deactivating app-inst:salso had run on DSP type cores. This sub-step in placing the activationapp-inst:s to their target core slots uses as one of its inputs the newand preceding versions of (the core slot ID indexed) app-inst ID andcore type arrays per Table 5, to allow matching activating app-inst:sand the available core slots according to the core type.

Architectural Cost-Efficiency Benefits

Advantages of the system capacity utilization and applicationperformance optimization techniques described in the foregoing include:

-   -   Increased user's utility, measured as demanded-and-allocated        cores per unit cost, as well as, in most cases, allocated cores        per unit cost    -   Increased revenue generating capability for the service provider        from CE based billables, per unit cost for a system 1. This        enables increasing the service provider's operating cash flows        generated or supported by a system 1 of certain cost level.        Also, compared to a given computing service provider's revenue        level, this reduces the provider's cost of revenue, allowing the        provider to offer more competitive contract pricing, by passing        on at least a portion of the savings to the customers (also        referred to as users) running programs 620 on the system 1,        thereby further increasing the customer's utility of the        computing service subscribed to (in terms of compute capacity        received when needed, specifically, number of cores allocated        and utilized for parallel program execution) per unit cost of        the service.

At a more technical level, the dynamic parallel processing techniquesper FIGS. 1-10 allow cost-efficiently sharing a manycore based computinghardware among a number of application software programs, each executingon a time variable, dynamically optimized number of cores, maximizingthe whole system data processing throughput, while providingdeterministic minimum system processing capacity access levels for eachof the applications configured to run on the given system.

Moreover, the hardware operating system 540 and the processing fabricmemory access subsystem 800 (described in relation to FIGS. 5-10)enables running any application task on a processor 500 at any of itscores at any given time, in a restriction free manner, with minimizedoverhead, including minimized core idle times, and without a need for acollective operating system software during the system runtime operation(i.e., after its startup or maintenance configuration periods) to handlematters such as monitoring, prioritizing, scheduling, placing andpolicing user applications and their tasks. The hardware OS 540 fabricmemory access subsystem 800 achieve this optimally flexible use of thecores of the system in a (both software and hardware) implementationefficient manner (including logic and wiring resource efficiently),without a need for core to core level cross-connectivity, as well asmemory efficiently without a need for the cores to hold more than oneapp-task-inst's processing state (if any needed) within their memoriesat a time. Instead of needing core to core cross-connects for inter-taskcommunications and/or memory image transfers, the memory accesssubsystem 800 achieves their purposes by more efficiently (in terms ofsystem resources needed) through a set of mux:s connecting the coreswith appropriate app-task-inst specific memory segments at the fabricmemory arrays. The system 1 architecture enables application tasksrunning on any core of any processing stage of the system to communicatewith any other task of the given application without requiring any suchcommunicating tasks to know whether and where (at which core) any othertask is running at any given time. The system thus providesarchitecturally improved scalability for parallel data processingsystems as the number of cores, applications and tasks withinapplications grows.

To summarize, the dynamic parallel execution environment provided by thesystem 1 enables each application program to dynamically get a maximizednumber of cores that it can utilize concurrently so long as suchdemand-driven core allocation allows all applications on the system toget at least up to their entitled number of cores whenever theirprocessing load actually so demands.

The presented architecture moreover provides straightforward IO as wellas inter-app-task communications for the set of application (server)programs configured to run on the system per FIG. 1. The external worldis typically exposed, for any given one of such applications, with avirtual singular app-instance instance (proxy), while the systemsupports executing concurrently any number of instances of any givenapp-task on the core fabrics 510 of the processing stages 300 (withinthe limit of core slot capacity of the system).

To achieve this, the architecture involves an entry-stage(“master-stage”) processing system (typically with the master tasks ofthe set of applications 610 hosted on it), which distribute the receiveddata processing workloads for worker-stage processing systems, whichhost the rest of the tasks of the application programs, with theexception of the parts (tasks) of the program hosted on the exit stageprocessing system, which typically assembles the processing results fromthe worker stage tasks for transmission to the appropriate externalparties. External users and applications communicates directly with theentry and (in their receive direction, exit) stage processing systemi.e. with the master tasks of each application, and these master taskspass on data load units (requests/messages/files/steams) for processingby the worker tasks on the worker-stage processing systems, with eachsuch data unit identified by their app-task instance ID#s, and with theapp ID# bits inserted by controllers 540, to ensure inter-taskcommunications stay within their authorized scope, by default within thelocal application. There may be multiple instances of any given (locallyhosted) app-task executing simultaneously on both the entry/exit as wellas worker stage manycore processors, to accommodate variations in thetypes and volumes of the processing workloads at any given time, bothbetween and within the applications 620 (FIG. 6).

The received and buffered data loads to be processed drive, at least inpart, the dynamic allocating and assignment of cores among theapp-inst:s at any given stage of processing by the multi-stage manycoreprocessing system, in order to maximize the total (value adding, e.g.revenue-generating) on-time IO data processing throughput of the systemacross all the applications on the system.

The architecture provides a straightforward way for the hostedapplications to access and exchange their IO and inter-task data withoutconcern of through which input/output ports any given IO data units mayhave been received or are to be transmitted at any given stage ofprocessing, or whether or at which cores of their host processors anygiven source or destination app-task instances may be executing at anygiven time. External parties (e.g. client programs) interacting with the(server) application programs hosted on the system 1 are likewise ableto transact with such applications through a virtual static contactpoint, i.e., the (initially non-specific, and subsequently specifiableinstance of the) master task of any given application, while within thesystem the applications are dynamically parallelized and/or pipelined,with their app-task instances able to activate, deactivate and belocated without restrictions.

The dynamic parallel program execution techniques thus enabledynamically optimizing the allocation of parallel processing capacityamong a number of concurrently running application software programs, ina manner that is adaptive to realtime processing loads of theapplications, with minimized system (hardware and software) overheadcosts. Furthermore, the system per FIGS. 1-10 and related descriptionsenable maximizing the overall utility computing cost-efficiency.Accordingly, benefits of the application load adaptive, minimizedoverhead multi-user parallel data processing system include:

-   -   Practically all the application processing time of all the cores        across the system is made available to the user applications, as        there is no need for a common system software to run on the        system (e.g. to perform on the cores traditional system software        tasks such as time tick processing, serving interrupts,        scheduling, placing applications and their tasks to the cores,        billing, policing, etc.).    -   The application programs do not experience any considerable        delays in ever waiting access to their (e.g. contract-based)        entitled share of the system processing capacity, as any number        of the processing applications configured for the system can run        on the system concurrently, with a dynamically optimized number        of parallel (incl. pipelined) cores allocated per an        application.    -   The allocation of the processing time across all the cores of        the system among the application programs sharing the system is        adaptive to realtime processing loads of these applications.    -   There is inherent security (including, where desired, isolation)        between the individual processing applications in the system, as        each application resides in its dedicated (logical) segments of        the system memories, and can safely use the shared processing        system effectively as if it was the sole application running on        it. This hardware based security among the application programs        and tasks sharing the manycore data processing system per FIGS.        1-10 further facilitates more straightforward, cost-efficient        and faster development and testing of applications and tasks to        run on such systems, as undesired interactions between the        different user application programs can be disabled already at        the system hardware resource access level.        The dynamic parallel execution techniques thus enable maximizing        data processing throughput per unit cost across all the user        applications configured to run on the shared multi-stage        manycore processing system.

The presented manycore processor architecture with hardware basedscheduling and context switching accordingly ensures that any givenapplication gets at least its entitled share of the dynamically sharedparallel processing system capacity whenever the given applicationactually is able to utilize at least its entitled quota of systemcapacity, and as much processing capacity beyond its entitled quota asis possible without blocking the access to the entitled and fair shareof the processing capacity by any other application program that isactually able at that time to utilize such capacity that it is entitledto. For instance, the dynamic parallel execution architecture presentedthus enables any given user application to get access to the fullprocessing capacity of the manycore system whenever the givenapplication is the sole application offering processing load for theshared manycore system. In effect, the techniques per FIGS. 1-10 providefor each user application with an assured access to its contract basedpercentage (e.g. 10%) of the manycore system throughput capacity, plusmost of the time much greater share, even 100%, of the processing systemcapacity, with the cost base for any given user application beinglargely defined by only its committed access percentage worth of theshared manycore processing system costs.

The references [1], [2], [3], [4], [5], [6], [7], [8] and [9] providefurther reference specifications and use cases for aspects andembodiments of the invented techniques. Among other such aspectsdisclosed in these references, the reference [4], at its paragraphs69-81 and its FIG. 11, provides descriptions for a billing subsystem1100 (see FIG. 7 herein for context) of a controller 540 of a manycoreprocessing system 500 according to an embodiment of the invention.

This description and drawings are included to illustrate architectureand operation of practical and illustrative example embodiments of theinvention, but are not meant to limit the scope of the invention. Forinstance, even though the description does specify certain systemparameters to certain types and values, persons of skill in the art willrealize, in view of this description, that any design utilizing thearchitectural or operational principles of the disclosed systems andmethods, with any set of practical types and values for the systemparameters, is within the scope of the invention. For instance, in viewof this description, persons of skill in the art will understand thatthe disclosed architecture sets no actual limit for the number of coresin a given system, or for the maximum number of applications or tasks toexecute concurrently. Moreover, the system elements and process steps,though shown as distinct to clarify the illustration and thedescription, can in various embodiments be merged or combined with otherelements, or further subdivided and rearranged, etc., without departingfrom the spirit and scope of the invention. It will also be obvious toimplement the systems and methods disclosed herein using variouscombinations of software and hardware. Finally, persons of skill in theart will realize that various embodiments of the invention can usedifferent nomenclature and terminology to describe the system elements,process phases etc. technical concepts in their respectiveimplementations. Generally, from this description many variants will beunderstood by one skilled in the art that are yet encompassed by thespirit and scope of the invention.

What is claimed is:
 1. A system comprising a plurality of subsystems,implemented on processors, for coordinating resource management of apool of processing resources on behalf of application programs, thesystem comprising: a first subsystem configured to, for each period of aplurality of allocation periods, allocate a first number of a pluralityof cores of the pool of processing resources to one or more of theplurality of application programs based, for each of the one or moreapplication programs, on a processing demand of the respectiveapplication program up to a respective processing resource quota,wherein the processing demand is based at least in part on a count ofready-to-execute instances for the respective application program, andallocate a second number of the plurality of cores after allocating thefirst number of the plurality of cores to one or more programs of theplurality of application programs based in part on remaining processingdemand of each of the plurality of application programs, such thatallocations of the plurality of cores are adjusted over time based atleast in part on changes in the processing demands of each of theplurality of application programs; a second subsystem configured to, foreach of the plurality of application programs, select a number ofhighest priority instances, corresponding to a number of cores allocatedto the respective program by the first subsystem, of one or moreinstances of the respective program for a next allocation period of theplurality of allocation periods, wherein the number of highest priorityinstances are selected based at least in part on a relative readinessfor execution among one or more instances of the respective program,wherein any of the ready-to-execute instances of the respective programhas a higher selection priority than any other instances of therespective program; and a third subsystem configured to assign thenumber of highest priority instances of each of the plurality ofapplication programs selected by the second subsystem to the cores thatare allocated to the corresponding program during the next allocationperiod of the plurality of allocation periods.
 2. The system of claim 1,wherein a length of each allocation period of the plurality ofallocation periods is set to a same predetermined interval.
 3. Thesystem of claim 1, wherein the pool of processing resources comprises aplurality of central processing units (CPUs).
 4. The system of claim 1,wherein at least a subset of the plurality of cores are virtual cores.5. The system of claim 1, wherein allocating the plurality of cores bythe first subsystem comprises allocating the plurality of cores based atleast in part on respective volumes of queued data available to each ofthe plurality of application programs for processing.
 6. The system ofclaim 5, wherein selecting the instances by the second subsystemcomprises, for each program of the plurality of application programs,prioritizing the respective one or more instances in an executionpriority order based at least in part on a number of items of queueddata within an input data buffer associated with each instance.
 7. Thesystem of claim 5, wherein the volumes of queued data, for each programof one or more of the plurality of application programs, includemessages associated with one or more specific instances of the one ormore instances of the respective program.
 8. The system of claim 1,wherein the ready-to-execute instances include at least one of a numberof assigned instances currently assigned to a given core of theplurality of cores, and a number of unassigned instances currentlyunassigned to a given core of the plurality of cores.
 9. The system ofclaim 1, wherein the processing resource quota comprises an entitlementheld by the respective application program.
 10. The system of claim 1,wherein at least one of the first subsystem, the second subsystem, andthe third subsystem comprises a plurality of instructions, stored to anon-transitory computer readable medium which, when executed oncircuitry, perform a subset of the operations of the respectivesubsystem.
 11. The system of claim 1, wherein at least one of the firstsubsystem, the second subsystem, and the third subsystem comprisesdedicated hardware for performing a subset of the operations of therespective subsystem.
 12. A system comprising a plurality of subsystems,implemented on processors, for coordinating resource management of apool of processing resources on behalf of application programs, thesystem comprising: a first subsystem configured to periodically allocatea plurality of cores of the pool of processing resources to a pluralityof application programs based at least in part on i) respectiveexpressions of demand for processing resources of the plurality ofprocessing cores by each of the plurality of application programs, andii) respective entitlements to the processing resources of the pluralityof processing cores, wherein a respective entitlement is held by each ofone or more programs of the plurality of application programs; andwherein the respective expression of demand is based at least in part ona count of ready-to-execute instances of each of the plurality ofapplication programs; a second subsystem configured to, for each programof the plurality of application programs, select a number of highestpriority instances, corresponding to a number of cores allocated to therespective program, of one or more instances of the respective programfor a next allocation period, wherein each of the number of highestpriority instances is selected based at least in part on a relativereadiness for execution among the one or more instances of therespective program; wherein any of the ready-to-execute instances of therespective program has a higher selection priority than any otherinstances of the respective program; and a third subsystem configured toassign the number of the highest priority instances of each of theplurality of application programs for execution on a respective one ormore cores of the cores that are allocated to the corresponding program;wherein allocating the plurality of cores by the first subsystemcomprises allocating a first number of the plurality of cores to thehighest priority instances of the one or more programs based, for eachof the one or more programs, on respective expression of demand up tothe respective entitlement, and after allocating the first number,allocating a second number of the plurality of cores to a second numberof instances of one or more of the plurality of application programsbased at least in part on an unmet portion of the expression of demandof the respective application program after allocating the first number.13. The system of claim 12, wherein: the system schedules the allocatingperiodically, once per core allocation period, wherein the coreallocation period is a predetermined period of time; and the allocating,the selecting, and the assigning are performed within each predeterminedcore allocation period.
 14. The system of claim 13, wherein assigningthe number of highest priority instances of each of the plurality ofapplication programs to the plurality of cores by the third subsystemcomprises preserving core assignments to instances selected forexecution on consecutive core allocation periods.
 15. The system ofclaim 13, wherein the respective expressions of demand of each of theplurality of application programs is based at least in part on a volumeof data available for processing by the respective application program.16. The system of claim 15, wherein the respective volume of dataavailable for processing by each application program comprises a numberof processing data units queued to at least one buffer associated withthe respective application program.
 17. The system of claim 12, whereinthe selecting by the second subsystem comprises, for each of theplurality of application programs, prioritizing the respective one ormore instances based at least in part on a respective volume of dataavailable for processing by each instance of the respective one or moreinstances.
 18. The system of claim 12, wherein selecting the instancesof each of the plurality of application programs by the second subsystemcomprises ranking the instances of each of the plurality of applicationprograms in an execution priority order.
 19. The system of claim 12,wherein one or more of the plurality of cores are virtual processingcores.
 20. The system of claim 12, wherein at least one of the firstsubsystem, the second subsystem, and the third subsystem comprisesdedicated hardware for performing a subset of the operations of therespective subsystem.
 21. The system of claim 12, wherein at least oneof the first subsystem, the second subsystem, and the third subsystemcomprises a plurality of instructions, stored to a non-transitorycomputer readable medium which, when executed on circuitry, perform asubset of the operations of the respective subsystem.
 22. A method formanaging processing resources of a parallel processing system on behalfof application programs, the method comprising: identifying, by acontrol system, a respective expression of demand for computing capacityof the processing resources of the parallel processing system associatedwith each of one or more programs of a plurality of applicationprograms; identifying, by the control system, a respective entitlementfor the computing capacity associated with each of one or more programsof the plurality of application programs; wherein the respectiveexpression of demand is based at least in part on a count ofready-to-execute instances of each of the plurality of applicationprograms; allocating, by the control system, a plurality of units of thecomputing capacity of the parallel processing system to a subset of theplurality of application programs, wherein the allocating is based atleast in part on i) the expressions for demand of each of the subset ofapplication programs and ii) the entitlements of the one or moreprograms of the subset of application programs; for each program of thesubset of application programs, assessing, by the control system, arelative readiness for execution among one or more instances of therespective program, and selecting, by the control system, a number ofhighest priority instances of the one or more instances of therespective program for execution on the parallel processing systemaccording to a) a count of units of the plurality of units of computingcapacity allocated to the respective program, and b) the relativereadiness for execution among the one or more instances of therespective program; wherein any of the ready-to-execute instances of therespective program has a higher selection priority than any otherinstances of the respective program; and assigning the number of thehighest priority instances of each of the plurality of applicationprograms for execution on a respective one or more units that areallocated to the corresponding program; wherein, for each program of theone or more application programs having a respective expression ofdemand meeting or exceeding a respective entitlement, allocatingcomprises allocating at least a number of units of computing capacityadequate to meet the entitlement of the respective program; and wherein,for at least one program of the plurality of application programs,allocating comprises allocating at least one unit of computing capacityexceeding the entitlement of the respective program.
 23. The method ofclaim 22, wherein the control system comprises dedicated hardware logicfor performing at least a portion of the operations of the controlsystem.
 24. The method of claim 22, wherein assessing the relativereadiness for execution comprises ranking the one or more instances ofthe respective program according to an execution priority order.
 25. Themethod of claim 22, wherein ranking is based at least in part on arespective volume of data available to each instance of the one or moreinstances for processing.
 26. The method of claim 22, wherein theexpression of demand is based at least in part on a respective volume ofdata available to each program of the plurality of application programsfor processing.
 27. The method of claim 22, wherein assessing relativereadiness for execution comprises assessing, for each instance of theone or more instances of the respective program, availability of data tothe respective instance for processing.
 28. The method of claim 22,further comprising repeating, for each of a plurality of core allocationperiods, the identifying the respective expression of demand, theallocating, the assessing, the selecting, and the assigning.
 29. Themethod of claim 28, wherein each of the plurality of core allocationperiods is a predetermined period of time.
 30. The method of claim 28,wherein the assigning comprises assigning for execution during a nextperiod of the plurality of core allocation periods.